Comparison of Two Contextual Post - Processing Algorithms for Text Recog

The binary n-gra~ and Viterbi algorithms are alternative approaches to con-te.xtual post-processing for text produced by a noisy channel such as an optical character recognizer. The paper describes the underlying theory of each approach in unified terminology, presents a storage efficient data structure for the binary n-gram algorithm and a recursive formulation for the viterbi algorithm. Relative merits of the two methods based on extensive experiments with each algorithm are given. 1. INTRODUCTION A number of programs for utilizing some form of contextual knowledge for post-processing of garbled text produced by a noisy channel such as an optical Character Recognition [OCRJ machine are described in the literature. Among these one can discern two different types of representation of context: statistical and structural. This paper describes our experiences with two contextual post-processing algorithms which are based on each of the two different types of representation. Statistical representation of context consists of models of word generation processes , e.g., Markovian model of text source, whose origins are in information theory. A particular algorithm of this class that we consider is the Viterbi al-od thm [3J and its modifications 11,6] • The structural approach, which is closer to the artificial intelligence framework, is based on a deterministic representation of context; examples are implicit or explicit representations of a lexicon, syntax or semantics. The method of this class we consider is the binary n-gram algorithm (2,5] which represents the syntax of a lexicon in the form of a set of binary arrays. 2. BINARY n-GRAM ALGORITHM